INITIATION AND SUPPORT OF VIDEO CONFERENCING USING INSTANT 



MESSAGING 

Related Applications 

This application is based on U. S. Provisional Patent Application serial number 
60/250,695 filed on December 1, 2000, the contents of which are incorporated herein in their 
entirety. 

Background of the Invention 

Video conferencing via networks such as the Internet is becoming increasingly popular. 
Video conferences on the Internet are typically supported by a video conferencing server such as 
the Multipoint Conference Unit (MCU), manufactured, sold and licensed by First Virtual 
Communications, Inc. of Nashua, New Hampshire and Santa Clara, California. Participants in a 
video conference supported by such servers communicate via their local video conferencing 
client software over the Internet with the MCU, which provides the audio and/or video 
conferencing data to some or all of the participants. Typically, video conferences are initiated by 
one or more prospective participants communicating with the MCU, such as by visiting a 
dedicated website, to identify invitees to the conference. The MCU then initiates and supports 
the video conference by providing the appropriate conference data to the client associated with 
each of the identified participants through network links with each of the clients. 

Instant messaging (EM) is becoming an increasingly popular utility on networks such as 
the Internet. Under instant messaging, IM clients on a network can communicate with each other 
via an IM server or messenger service of the network. The messenger service also typically 
provides a "presence" feature which permits IM clients to know the status of other clients on the 
network serviced by the messenger service. The status information for a client includes whether 
the client is online and available to receive instant messages from other clients and whether the 
client presently at his workstation, which can be determined by the level of activity at the client's 
workstation. The presence feature also allows clients to declare themselves unavailable to 
receive instant messages so that incoming messages can be blocked. 
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Because of the wide range of capabilities and the convenience of instant messaging, it is 
becoming a popular means of communication among clients on networks. For example, 
individual companies are using instant messaging utilities as their preferred means of 
communication among employees. 

Summary of the Invention 

The present invention is directed to a system and method for combining the convenience 
of instant messaging with video conferencing capability to allow a user to initiate a video 
conference using IM. The system of the invention includes at least two client nodes which are 
adapted to communicate with each other via an instant messaging utility. An instant messaging 
server supports instant messages between the two client nodes. A second server supports video 
conferences between video conference participants. In accordance with the invention, a video 
conference is initiated between video conference participants in response to an instant message 
transmitted between the at least two client nodes. 

The video conference participants can participate in the video conference by any of 
several possible communication modes. For example, a participant may participate in a video 
conference via a desktop computer or dedicated video conferencing system using a video 
conferencing standard protocol such as H.323 video conferencing protocol. The participants may 
communicate via the public switched telephone network (PSTN), cellular communication, and 
ISDN protocol or an ATM protocol. Each participant may communicate via one or more internet 
IP gateways which form the interface between the internet and a particular participant's 
communication mode. 

In one embodiment, the instant message contains information used to identify the 
communication mode to be used by a participant in the video conference. That is, an IM client 
with IM capability can initiate a video conference using the IM utility to identify participants in a 
video conference. A database used in connection with initiating video conferences indexes and 
catalogs communication and/or contact data for each participant in the identified video 
conference. This information is used to set up the video conference among the identified 
participants using each participant's preferred communication mode. These communication 
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modes can be any of those identified above. In one embodiment, another server is used to 
coordinate the video conference among the participants identified by the instant message. This 
third server, in conjunction with the database, contains all of the communication mode 
information mentioned above and tabulates the communication mode information with 
participants in each initiated video conference. This third server can receive communication 
mode information from the instant message used to initiate the video conference. The 
information can then be used by the third server to set up the video conference using the 
appropriate communication modes associated with the identified participants. 

Hence, in accordance with the invention, the increasingly popular instant messaging 
utility implemented by many internet service providers (ISP) such as AOL, MSN, etc. can be 
used to automatically initiate a video conference. This can be a significant convenience to 
business and personal users compared to conventional systems in which one may have to visit a 
dedicated website to initiate a video conference. 

Brief Description of the Drawings 

The foregoing and other objects, features and advantages of the invention will be apparent 
from the more particular description of preferred embodiments of the invention, as illustrated in 
the accompanying drawings in which like reference characters refer to the same parts throughout 
the different views. The drawings are not necessarily to scale, emphasis instead being placed 
upon illustrating the principles of the invention. 

FIG. 1 contains a schematic block diagram illustrating one embodiment of the system of 
the invention in which an instant messaging utility is used to initiate and support a video 
conference. 

FIGs. 2 and 3 are functional block diagrams and protocol diagrams, respectively, of 
initiation of a video conference in which a first instant messaging client invites a second instant 
messaging client to participate in the conference, in accordance with one embodiment of the 
invention. 

FIGs. 4 and 5 are functional block diagrams and protocol diagrams, respectively, of 
initiation of a video conference in which a first client initiates the conference via a H.32x 
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communication to an instant messaging client, in accordance with one embodiment of the 
invention. 

FIG. 6 is a protocol diagram for the scenario in which M users call H.32x users, in 
accordance with one embodiment of the invention. 

FIG. 7 contains a functional block diagram of the operation of the invention with an IM 
client, in accordance with one embodiment of the invention. 

FIG. 8 illustrates the event sequence between an IM client and a remote buddy who does 
not have the plugin, in accordance with one embodiment of the invention. 

FIG. 9 illustrates the event sequence between two plugins, in accordance with one 
embodiment of the invention. 

FIG. 10 contains a schematic functional block diagram of integrating the plugin of the 
invention with an IM application using vendor APIs. 

Detailed Description of Preferred Embodiments of the Invention 

FIG. 1 contains a schematic block diagram illustrating one embodiment of the system 10 
of the invention in which an instant messaging utility is used to initiate and support a video 
conference. The system 1 0 is implemented over a network such as the Internet, and 
communication among the system components is via network protocol such as the Internet 
Protocol (IP) indicated by the solid lines with arrowheads in FIG. 1 . 

The system includes a number of clients 12 which are capable of communicating over the 
network by some conventional instant messaging (IM) utility. Each client 12 can be, for 
example, a computer adapted to communicate over the Internet. Each of the clients 12 includes 
an IM client software module 16 which permits the client 12 to send and receive instant 
messages. Each of the IM client modules 16 can create, transmit and receive instant messages. 
The IM utility is supported by a messenger service or IM server 14 on the network. An IM client 
12/16 creates an instant message and forwards the message to the IM server 14. The IM server 
14 analyzes the message and forwards it to its intended recipient IM client 12/16. 

In accordance with the invention, the system 10 also provides the capability to perform 
video conferencing over the network, i.e., Internet. Video conferences among video 
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conferencing (VC) clients 28 are initiated and supported by a video conferencing server 24, also 
referred to herein as a multipoint conferencing unit (MCU). The MCU can be of the type 
manufactured, sold and licensed by First Virtual Communications, Inc. of Nashua, New 
Hampshire and Santa Clara, California, the assignee of the present patent application. When a 
5 video conference is in session, the MCU 24 enables and manages the conference. The MCU 24 
formats audio and video data from the clients 28 and sends the audio and/or video data to the 
clients 28 as participants in the conference. The MCU can receive and transmit the video 
conference data to and from the clients 28 using some standard video conferencing protocol, for 
example, H.323 protocol, T.120 protocol, session initiation protocol (SIP) or CUseeMe protocol 
10 provided by First Virtual Communications, Inc., the present assignee. 

As illustrated in FIG. 1, in general, the MCU 24 can support video conferences with VC 
clients 28 communicating by various means. For example, VC clients 28 can participate in a 
^ conference via the public switched telephone network (PSTN), or via an ISDN H.320 standard 
£3 protocol, or an ATM H.321 standard protocol, or a cellular communication protocol, etc. To 
15 %j support each of these means of communication, the MCU 24 interfaces with such clients via a 
JjY gateway 26 which provides an interface which translates between the protocol used by the VC 
€! client 28 participating in the video conference and the MCU 24 supporting the video conference. 

|,t Also, the VC client 28D can communicate directly with the MCU 24 in a video conferencing 

f \ \ 

*2j protocol, eliminating the need for a gateway 26. 

20 y Conventionally, video conferences are initiated on the MCU 24 by one of the VC clients 

§4 identifying the "invitees" or participants in a video conference such as by visiting a web page 
supported by the MCU 24. In response, the MCU 24 contacts each of the participants and sets up 
communication links with them or their associated gateways using a video conferencing protocol. 
When the conference begins, the MCU 24 transmits and receives conferencing data to and from 

25 the participating clients. The conference can be initiated using a server, referred to herein as a 
"click-to-meet" CTM server 20, which supports the web page at which the video conference is 
initiated by one of the VC clients identifying the participants. The CTM server 20 interfaces 
with a database 22 which stores the identities and contact information for all potential 
participants in a conference. The database 22 also stores the communication modes used by each 
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participant. When the participants are identified, the server 20 uses the database 22 to set up the 
video conference using the communication modes for all of the identified participants. 

In accordance with the invention, video conferences are initiated using the instant 
messaging utility. To that end, each IM client 12 includes a video conference module 18 which 
interfaces with both the IM client module 16 and the CTM server 20 to initiate a video 
conference. In accordance with the invention, the IM utility on each client 12 is modified to 
permit a client to identify invitees to a conference. These invitees can be, for example, taken 
from a buddy list supported by the IM utility. In one embodiment, a user at an IM client 12 can 
set up a video conference by clicking on one or more buddies in a buddy list provided by the IM 
utility. 

The IM client module 16 and the video conference module 18 within each client 12 are 
interface to each other. When an instant message setting up a video conference is sent by a client 
12 via its IM client module 16, the video conference module 18 detects the message. It extracts 
information from the message which is used to set up the video conference automatically through 
the CTM server 20. The information extracted from the IM can include the identity, address and 
communication mode for each prospective participant in the conference. Once the CTM server 
20 receives this information, it can initiate the conference via the MCU 24. Alternatively, the 
information extracted from the instant message can include the identity of the prospective 
participants in the video conference. This identity information is provided to the CTM server 20 
which can associate each participant with its address, communication mode, gateway address, 
and other required information, using the database 22. This information is used in setting up the 
video conference in the MCU 24. 

FIG. 1 is a generalized block diagram which illustrates the general case of the invention, 
namely, one or more IM clients 12 being capable of automatically initiating a video conference 
among one or more video conferencing clients 28 using an IM utility. It will be understood, 
however, that in one likely configuration of the invention, the IM clients 12 are the same as the 
VC clients 28. For example, each IM client 12 can be a user's computer adapted to communicate 
on the Internet. The computer can have the IM client module 16 and the video conferencing 
client 18 installed. Also, the computer can include the video conferencing client module as well 
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In this configuration, the user can decide to initiate a video conference on his own computer. He 
can use his instant messaging facility to invite other participants. In response, the CTM server 20 
and MCU 24 can initiate the video conference on the inviter's computer and with the other 
invited participants. 

5 In general, the instant-message-initiated video conferencing capability of the invention is 

implemented in modules at the individual clients. For example, the clients 12 which initiate the 
conference via instant messages include both IM modules 16 responsible for sending and 
receiving instant messages and video conference initiation modules 18 which extract data from 
the instant messages to initiate the conferences, in accordance with the foregoing description. 
10 The video conferencing clients 28 include the actual video conferencing client modules which 
interface with the MCU 24 to present the video conference to the users at the clients 28. To 
participate in a video conference, the client 28 must have the video conference module installed. 
In the event that an invitee to a conference does not have the module installed, the client 28 is 
0 automatically directed to an installation center which provides the necessary modules in plug-in 
15 %t form via the Internet. 

% In accordance with the invention, the CTM server 20, in conjunction with the database 

# 22, maintains the capability to initiate video conferences with the various VC clients 28. As 

ra. 

U noted above, the database 22 and CTM server 20 maintain information as to the modes of 

hi . . 

%l communication, e.g., telephone, cellular, etc., used by each client 28 such that, when a video 
20 W conference is initiate, each client 28 can be contacted and connected to the conference by the 
%t correct communication mode. In accordance with the invention, the CTM server 20 also 
provides an additional capability in that it "advertises" or makes information available as to 
presence of the clients 28. That is, analogous to the method by which an instant messaging 
system advertises presence of individual IM clients on the network, the CTM server 20 in 
25 accordance with the invention advertises presence of individual VC clients 28. As noted above, 
these VC clients 28 may be colocated with corresponding IM clients 12, such as the case where a 
user's desktop computer contains an IM client software module 16, a video conference initiation 
software module 18 and a video conferencing client software module 28. As an example of the 
CTM server's capability to advertise presence of VC clients 28, the CTM server 20 may 
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advertise that a particular client 28 is available to receive a cellular communication. Also, the 
CTM server 20 and database 22 may be set up, for example, to indicate a client 28 as being 
reachable by cellular communication when the clients' IM presence indicates that he is away 
from his computer. This presence feature is implemented via a link between the CTM server 20 
and the IM server 14. As a result, IM clients 12 can know whether and by what means of 
communication particular clients 28 are available for conferencing. 

In accordance with an embodiment of the invention, there are generally three methods of 
initiating a video conference. The messaging in each case is basically the same and is in 
accordance with the following description, but the endpoints, i.e., VC clients 28, may either call 
on to the conference or be invited into the conference. Any conference can be made up of either 
of these caller/callee types. 

The first initiation method is referred to as desktop initiation. Instant messaging is used 
via menu choice, added buttons or other means of selection to identify participants. The caller 
initiates invitation through IM and launches either a local application (such as Microsoft 
NetMeeting or CUseeMe client software provided by First Virtual Communications, Inc., the 
present assignee), a web page with a plugin (such as CUseeMe Web plugin provided by First 
Virtual Communications, Inc.) or its own embedded audio/video technology, or is called on its 
local computer or separate device, e.g., telephone, dedicated room video conferencing system, 
etc. 

The second method involves IM presence advertised by the CTM server 20. Presence of 
devices such as room systems, cellular phones, etc., is advertised by the CTM server on behalf of 
users available at cell phones, room systems, etc.. In this case, instant messages may not be 
exchanged with the actual device used to participate in the conference. 

In the third method, presence is advertised for an existing conference. For example, the 
existing conference, such as a regularly scheduled company meeting, is listed on an IM client 12 
as a "Buddy." The client 12 can join the conference simply by clicking on the buddy icon for the 
conference. 

The following describes the modules at the clients used to initiate and support video 
conferences using the instant messaging utility, in accordance with the invention. 
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1. Components 

The components of the system of the invention which implements audio/video enhanced 
instant messaging and video conferencing initiated using the instant messaging system will now 
be described. 



LI Desktop Client 

The desktop client has a user interface similar to that of a traditional instant messaging 
client such as MSN and iPulse. It has the standard instant messaging features of contact list and 
its management functions such as add/remove group/contact, and an icon in the Windows taskbar 
10 tray. In one embodiment, audio and video conferencing is integrated into the instant message and 
group conference scenarios for up to 4 participants. Once in a group conference, the creator of a 
conference may invite additional participants. 



" m manufacturer (OEM) because it provides the customizations listed below. The implementation 

p. w 

y j of these items provides the following features: 
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1.1.1 Partner/OEM Friendly User Interface (UP 

The UI is adaptable to being provided as a component to an original equipment 



25 




A. Pluggable Resources (Graphics) - The ability to easily at least partially change the 
"skin" of the application, with OEM specific graphics. 

B. Ad Space (Standard) in client - Ad space is provided in industry standard format 
similar to AIM, with clickthrough support. 

C. URL launched Web based Conference Pages - All text and A/V enhanced conferences 
can be launched in web pages instead of the typical application windows. Sample 
templates are provided for OEMs to add their own graphics. A default page is provided 
by if the OEM partner does not supply one. 



1.1.2. Installation 



The application has a standard installer such as a standard Windows installer. 
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1.2. IM Call Router 

The back end component bridges the M and A/V conferencing environments and 
controls management of connections in the A/V enhanced IM system. It is designed to support 
any IM system a well as any video conferencing protocol. The capabilities of the IM Call Router 
include the following. 

A. Handle requests for calls from A/V enabled IM endpoints to other A/V enabled IM 
endpoints. 

B. Handle incoming calls from videoconferencing environments such as session 
initiation protocol (SIP) and H.323, and map these requests to specific IM users. 

C. Handle outgoing calls from A/V enabled IM clients to the same videoconferencing 
endpoints as item B. 

D. Request A/V conferences and redirect the callers and callees to the appropriate 
conference. 

The IM Call Router has the following components. 

A. IM Plugin - Responsible for all communication with IM clients through the IM server. 

B. Conference Allocator Plugin - Responsible for all communication between the IM 
router and the conference allocator. 

C. Call Redirector Plugin - This component handles incoming and outgoing calls from/to 
other network protocols such as SIP and H.323 and is responsible for all communication 
between the IM Call Router and the non-M endpoints. 

Further details of the IM Call Router are described below. 

1.3. Smart Conference Allocator 

The smart conference allocator is responsible for creating conferences in an application 
service provider (ASP) video conferencing environment. The ASP environment in accordance 
with the invention can be of the type described in copending U. S. Patent Application serial 
number 09/735,828, filed on December 13, 2000, entitled, "Application Service Provider 



CSM-0002 



10 



Configuration and Method for a Conferencing Environment," assigned to First Virtual 
Communications, Inc., the assignee of the present application. The contents of that application 
are incorporated herein in their entirety. 

The ASP in this application also refers to a system configuration in which the system is 
installed into an enterprise which then deploys services only to its employees. This configuration 
is referred to herein as an enterprise internal deployment. 

2. A/V Enhanced Instant Messaging Scenarios 

There are two scenarios that are addressed in accordance with the invention. The two 
scenarios include IM users inviting other IM users to participate in a video conference and H.320 
standard protocol users calling IM users. For either scenario, the following items are provided in 
accordance with the invention. 

A. An intelligent server side component (IM Call Router) to handle conferencing 
requests on behalf of IM clients and/or the Intelligent H.323 call router. 

B. ASP components for dispatching incoming calls, dispatching and allocating 
conferences, and notifying the IM Call Router of certain events. 

C. An additional channel between the IM clients, through the IM server to the IM Call 
Router to communicate A/V conference requests and information. 

For all scenarios in this section, there are no client side APIs needed on the client side other than, 
for example, a CUseeMe Web software development kit (SDK) sold by First Virtual 
Communications, Inc. of Nashua, New Hampshire and Santa Clara, California and the additional 
IM channels for communication between IM components. The intelligence for finding 
conferences and other users stays on the server side and is implemented in the IM Call Router. 
The following sections describe the scenarios and components. 

2. 1 IM user Invites IM userfs) 

When one IM user invites another M user into an A/V conference, a private 
communications channel is opened through the IM Server to the IM Call Router. The IM Call 
Router initiates a conference via the conference allocator, with the appropriate attributes 
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determined by the capabilities of endpoints (determined from Presence info and M DB 
Properties). Once the conference information (IP, conference ID, potentially other conference 
attributes) is determined, the conference dispatcher initiates the location and invitation of the user 
through IM Call Router. Eventually the IM Call Router will get the conference information to 
the invitee(s) of which conference to connect to, and the connection will be initiated by the client 
through the CUseeMe Web SDK. FIGs. 2 and 3 are component and protocol diagrams of this 
process. 

2.2 Incoming H.32x Calls to IM Users 

In addition to IM users calling/conferencing with other IM users, an infrastructure for 
incoming and outgoing calls with h.32x systems is supported through the ASP and IM System. 
ASP components help map incoming h.32x calls to IM users, check their availability and 
through the IM system get them to meet the incoming caller in a conference allocated for an 
H.32x port. The mapping is done by additional destination h.32x alias information added to the 
initial call destination address to additional properties on user records in the IM database (i.e. 
El 64 alias, H323 ID). 

FIGs. 4 and 5 illustrate the components and protocol necessary to satisfy supporting 
incoming h.320 calls to IM users, with the added bonus of calls from other h.323 endpoints. 

2.3 Outgoing Calls to H.32x Users 

FIG. 6 is a protocol diagram for the scenario in which IM users call H.32x users. 

2.4 Conference Registered as a Buddy 

In this scenario, the backend components can register with the IM server a buddy or 
contact which is actually a conference, such that, when an IM endpoint initiates a message to that 
contact, the IM server can bring it into a video conference without any'special software on the 
desktop, unless it is needed for the actual video conferencing. All the call routing and 
information is figured out by the backend servers in this case. 
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The following describes the interaction between the IM client and the IM Server in 
accordance with the invention. The messages passed between the IM client and the IM server in 
initiating a video conference in accordance with the invention are described. 

L IM System Components 

This section describes the system components in accordance with the invention. 

1.1 The IM Client 

The IM Client is the software located at the client workstation. From a developer point of 
view this software has the following (low level) capabilities. 

A. Exchange standard IM messages with the IM Server 

B. Exchange custom IM messages with the IM Server 

C. Establish a connection with the MCU 

From a user point of view this software has the following (high level) capabilities. 

A. All the standard IM features 

B. Call another IM Client 

C. Call a H323 Client 

Examples IM Clients which can be used in connection with the invention include the IPulse 
application provided by Ericsson and Microsoft Windows XP Messenger. These can be used 
with the CUseeMe Web toolkit manufactured, sold and licensed by First Virtual 
Communications, Inc. of Nashua, New Hampshire and Santa Clara, California, which enables 
them to be audio and video capable. They can also be used with video capabilities included. 

1.2 The H323 Client 

In one configuration of the system of the invention, the H323 Client is standard H323 
software able to establish a connection to another H323 end point via a GateKeeper. 

1.3 The H323 Gate Keeper 

In one embodiment, the GateKeeper is a standard H323 GateKeeper able to communicate with 
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the IM Server and the MCU. 



1.4 TheMCU 

In one embodiment, the MCU is an H323 Multipoint Conferencing Unit (MCU) of the type sold 
by First Virtual Communications, Inc., the assignee of the present application. 

1.5 The IM Server 

In one embodiment, the IM Server is a piece of server software that is able to handle IM standard 
messages, some custom messages, and is also able to communicate with the call router and the 
GateKeeper. The IM Server software can be, for example, the IPulse server and some extension 
to it. Other examples of IM servers which can be used in connection with the invention include 
the Microsoft MSN and Exchange Servers. 

2. Calling Scenarios 

The following tables define the different calling scenarios that are supported by the IM 
System in accordance with the invention. 
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Table 1 . Case 1 : The Called party is not Connected. 



IM Caller 


IM Server 


IM Called Party 


Send a call request to the 
IM Server 

Call terminated 


Reply that the Callee is 
not connected 





Table 2. Case 2: The Called party is connected but refuses the call. 



IM Caller 


IM Server 


IM Called Party 


Send a call request to the 






IM Server 








Forward the call to the 






IM Called party 








The Client is ringing 






The Client rejects the call 






Tell the IM Server that the 






call was rejected by the 






user 




Forward the call 






rejection to the IM Caller 




Call terminated 
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Table 3, Case 3: The Called party is connected and accepts the call. 



IM Caller 


IM Server 


IM Called Party ! 


Send a call request to the 
IM Server 

Connect to the conference. 


Forward the call to the 
IM Called 

Create the conference. 
Send the conference 
address to both ends. 


The Client is ringing 

The Client accepts the call 

Tell the M Server that the 
user accepted the call. 

Connect to the conference. 



Table 4. Case 4: The Caller cancels the call while it rings. 



IM Caller 


IM Server 


IM Called 


Send a call request to the 






IM Server 


Forward the call to the 
IM Called 


The Client is ringing 


The Client tells the DM 






server that the user 






cancelled the call 


Forward the cancellation 
to the client. 


Call Terminated. 



CSM-0002 



16 



Parameters used in communication between the DM Caller (IM Client) and IM Server include the 
following. 



ry 



iConference: ConfAddress 

The iConference parameter contains the address of the conference the IM Client should 
connect to. The ConfAddress contains an IP address and an MPCS conference ID. 

iSettings: ConfSettings 

The iSettings parameter contains the parameters the IM Client should use to connect to 
the conference. A typical setting indicates the ability or inability of the client to send video. 



3. Messages 



7-7 

10 %j This section describes messages sent by the Client to the Server. 



3.1 Create Call 



il 'J 

p* This is the only synchronous message, all other messages are asynchronous an do not return 



k This message is sent by the caller client to the server when it wants to start a call. 

This is th 
anything. 



15 Parameters: 

iAddress : URL 

The iAddress parameter contains the address of the other end point to call. 

Reply: 

oError: Integer 

20 The oError return value contains the error code of the request. 

Possible error codes are: 
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ERR NO ERROR 

The Call was successfully created. 

ERRNOTALLOWED 

The IM Client is not allowed to create a call 

ERRBADADDRESS 

The IM Server does not recognize the address. 

oCallRef: Call Reference 

If the call was successfully created, oCallRef contains the address of the other end point 
to call. This reference is used by all other messages and remains valid until the call is 
terminated. 

3.2 Incoming Call 

This message is sent by the server to the called client after a call was created. 
Parameters: 

iCallRef: Call Reference 

The iCallRef parameter contains the unique reference to the call. 

iCaller: URL (optional) 

The iCaller parameter contains the address of the caller client. 

3.3 Terminated Call 

This message is sent by any client to the server when it cancels a pending call. The server 
also forwards this message to the other client. The Server itself can decide to send this message 
on its own. After this message the Call Reference is no longer valid 
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The IM call router will now be described in detail. The purpose of the IM Call Router is 
to let standard video conferencing clients, such as H.323, SIP, and CUseeMe protocol 
(manufactured, sold and licensed by First Virtual Communications, Inc., the present assignee) 
clients, call IM clients via a conference call hosted on a video conferencing server such as an 
MCU. 

1. Architecture 

The IM call router supports multiple IM systems and multiple video conferencing 
systems. In order to achieve this goal the IM call router has an extensible architecture that 
supports plugins. 

The following sections describe the different types of plugins supported by the IM call 

router. 

1.1 The Call Redirector Plugin 

The call redirector plugin provides an interface for video conferencing systems. It tells 
the IM call router that a call occurred, and then forwards the call status returned by the router to 
the video conferencing client. An IM call router may use multiple call redirector plugins 
simultaneously. The IM redirector plugin can be compatible with H323, SIP or CUseeMe 
protocols. 

1.2 The IM Plugin 

The IM plugin provides an interface for IM systems. It forwards the call received from 
the call redirector to the IM client, then returns the call status returned by the IM client to the IM 
call router. An IM call router may use multiple IM plugins simultaneously. The IM plugin can 
be Microsoft Windows Messenger IM, Ipulse, ICQ or AIM. 

1.3 The Conference Allocator Plugin 

The conference allocator plugin creates a video conference on the video conference server 
(MCU) that will host the IM client / video conferencing client session. An IM call router uses 
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only one allocator plugin. 

In one embodiment, the conference allocator may open a telnet connection with a video 
conferencing server. In another embodiment, the conference allocator performs load balancing, 
using a cloud of video conferencing servers. 

2. Plugins. APIs and Callbacks 

This section describes the APIs and callbacks that are implemented by the plugins. 
Because the IM call router may handle a lot of simultaneous calls, because it is time critical, and 
because some APIs may require user interaction, all the APIs are asynchronous, and none of 
them requires a reply. 

2. 1 Call Redirector IM Router Messages 

This section describes messages sent by the call redirector plugin to the IM router. 

2.1.1 CallStarted 

This message is sent by the call redirector when it receives a call. The call is then 
maintained in "ringing" mode until the IM router accepts or rejects the call. 

Parameters: 
iCallRef: String 

The iCallRef parameter is a string that contains a unique reference to the call. This 
reference is used by all the other messages and remains valid until the call is terminated. 

i Address: IMAddress 

The iAddress parameter contains the address of the IM client. The IMAddress type is a 

string. 

2.1.2 CallCanceled 

This message is sent by the plugin when the video conferencing client cancels a pending 
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call. 

Parameters: 
iCallRef: String 

The iCallRef parameter contains the unique reference to the call. This reference is the 
same as the one passed to the CallStarted message. 

2.2 IM Router to Call Redirector Messages 

This section describes messages sent by the IM router to the call redirector plugin. 
2.2.1 CallReiected 

This message is sent by the IM router in response to a CallStarted message when the call 
cannot be issued. This message terminates the call process, then the call reference is no longer 
valid. 

Parameters: 
iCallRef: String 

The iCallRef parameter contains the unique reference to the call. This reference is the 
same as the one passed to the CallStarted message. 

iError: Integer 

The iError parameter contains the reason why the call was rejected. 
Possible error codes are: 
ERRJ\OT_VALID 

The iAddress parameter passed to the CallStart message is not reconized by any IM plugin. 
ERR_NOT_FOUND 

The IM Client described in the iAddress parameter passed to the CallStart message was not 
found. 

ERRNOTCONNECTED 

The IM Client is offline. 
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ERR.BUSY 

The IM Client is busy. 
ERR REJECTED 
The IM Client refused the call. 
5 ERRTIMEOUT 

The IM Client did not answer the call. 

2.2.2 CallAccented 

This message is sent by the IM router in response to a CallStart message when the call is 
accepted. 

10 Parameters: 

ass iCallRef: String 

9 The iCallRef parameter contains the unique reference to the call This reference is the 

St same as the one passed to the CallStarted message. 

^ iConfTP: String 

1 5 M? The iConflP parameter contains the IP address of the video conferencing server (MCU) 

m that will host the conference. 

to*? 

hh iConfID: Integer 

The iConfID parameter contains the ED of the conference. 

2.3 IM Router To IM Plugin Messages 
20 This section describes messages sent by the IM router to the IM plugin. 

2.3.1 CallStarted 

This message is sent by the IM router when in receives an incoming call notification from 
the call redirector. When receiving this message, the IM plugin resolves the IM client address, 
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checks if the IM Client exists and is connected, and, if so, forwards the call request to the IM 
client, which will then pop up an "Incoming Call" dialog. 



Parameters: 
iCallRef: Merger 

The iCallRef parameter contains a unique reference to the call This reference is used by 
all the other messages and remains valid until the call is terminated. 

iAddress: IMAddress 

The iAddress parameter contains the address of the IM client. The IMAddress type is a 

string. 



10 w 2.3.2 CallCanceled 



This message is sent by the IM router when it receives a call cancelled notification from 
the call redirector. The IM plugin then forwards this notification to the IM client which will 
close the "Incoming Call" Dialog. 



|4 Parameters: 
15 % iCallRef: Integer 

hi 

W The iCallRef parameter contains the unique reference to the call. This reference is the 

|4 same as the one passed to the CallStarted message. 

2.4 IM Plugin To IM Router Messages 

This section describes messages sent by the IM plugin to the IM router. 

20 2.4.1 CallReiected 

This message is sent by the IM plugin in response to a CallStarted message when the call 
cannot be issued. This message terminates the call process, then the call reference is no longer 
valid. 
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Parameters: 
iCallRef: Integer 

The iCallRef parameter contains the unique reference to the call. This reference is the 
same as the one passed to the CallStarted message. 

iError: Integer 

The iError parameter contains the reason why the call was rejected. 
Possible error codes are: 
ERRNOTVALID 

The iAddress parameter passed to the CallStart message is not recognized by any IM plugin 
When the IM router receives this error code it attempts to use another IM plugin to resolve the 
address. If all the available IM plugins return this error code, the error is forwarded to the Call 
Redirector. 

ERRJVOTJFOUND 

The IM Client described in the iAddress parameter passed to the CallStart message was not 
found. 

ERR_NOT_CONNECTED 

The IM Client is offline. 

ERRBUSY 

The IM Client is busy. 

ERRREJECTED 

The IM Client refused the call. 

ERRJTIMEOUT 

The IM Client didn't answer the call. 

2.4.2 CallAccepted 

This message is sent by the IM plugin in response to a CallStart message when the call is 
accepted. When the IM router receives this call it asks the conference allocator to create a 
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conference then forwards the acceptance to the call redirector with the newly created conference 
address. 



Parameters: 
iCallRef Integer 

5 The iCallRef parameter contains the unique reference to the call. This reference is the 

same as the one passed to the CallStarted message. 

2.5 IM Router To Conference Allocator Messages 

This section describes the message sent by the IM router to the conference allocator. 



2.5.1 CreateConf 

10 « This message is sent by the IM router when the IM client accepted an incoming call. 



%4 Parameters: 

jpy iRequestRef: Integer 



^ The iRequestRef parameter is a unique reference that will help the IM router to identify 

l»* (Alf, please complete this sentence.) 

15 W iConfSettings: ConfSettings 

'\*k The iConfSettings parameter contains information about the conference. 



2.6 Conference Allocator to IM Router Messages 

This section describes messages sent by the conference allocator to the IM router. 
2.6.1 ConfCreated 

20 This message is sent by the conference allocator to tell the IM router that the conference 

was successfully created. 
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Parameters: 
iRequestRef: Integer 

The iRequestRef parameter is the unique request reference. This reference is the same as 
the one passed to the CreateConf message. 

iConflP : String 

The iConfDP parameter contains the IP address of the video conferencing server (MCU) 
that will host the conference. 

iConfID : Integer 

The iConfID parameter contains the ID of the conference. 

2.6.2 Failed 

This message is sent by the conference allocator to tell the IM router that it was not able 
to create a conference. 

Parameters: 
iRequestRef : Integer 

The iRequestRef parameter is the unique request reference. This reference is the same as 
the one passed to the CreateConf message. 

iError : Integer 

The iError parameter contains the ID of error that prevent the conference allocator to 
create the conference. 

The IM plugin for M systems in accordance with embodiments of the invention will now 
be described in detail. The plugin in accordance with the invention allows IM users to add 
video/audio capabilities to their IM client. It not only uses the user status and location features of 
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IM systems, but it also works together with the IM service. 

The EM plugin is composed of two distinct conceptual modules even if both can fit into 
one executable. The back-end application manages the low-level layer of the IM protocol and IM 
plugin user interface on the IM application's user interface. The conference room application 
5 uses CUseeMe Web technology, manufactured, sold and licensed by First Virtual 
Communications, Inc., the present assignee, to display the video window. 

The IM plugin in accordance with the invention is normally launched at system startup 
and runs in the background as an invisible application. A tray icon may be provided when the IM 
application is online if needed. 
10 In accordance with the invention, there are two approaches to integrating the plugin of the 

invention with a third party IM application. The first involves using a local proxy to monitor any 
IM protocol messages as well as send custom/private messages used for conference rendezvous 
pjj information, when no APIs are provided by the IM application vendor. The second involves 
G using the vendor-supplied APIs to monitor IM events and messages as well as send 

is;*? 

15 *y custom/private messages for conference rendezvous information. These may be done at either 

a : caller and/or callee, or only at the sever to invite endpoints like Windows XP Messenger, which 

*0 has a rendezvous protocol embedded. These two approaches are now described in detail. 

U 1. The Proxy 

W 

During installation, the proxy is inserted between the client and the IM server. The IM 
20 W plugin of the invention changes the IM application's connection settings to use a port on the local 
\& machine, where the proxy is listening. A solution with a real Socks proxy can also be 

implemented, changing the IM application's proxy settings instead of the IM server address 
itself. 

The proxy decodes the IM protocol. It is able to retrieve information on the buddies like 
25 IM ID, IP address, and status. IM data is forwarded to the IM server / IM client. 

FIG. 7 contains a functional block diagram of the operation of the invention with an 
existing third-party client. Incoming messages from other buddies are inspected and sometimes 
trapped in case of plugin-specific messages. Those messages can pass through the server, or 
come from a direct TCP connection with the buddies. Outgoing messages are intercepted as 
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well, allowing introduction of custom information inside the messages, which are then forwarded 
to the IM server. 

The plugin module is responsible for the integration of custom controls inside the third 
party M application's user interface that let the user start video connections. A click on the 
video button can trigger, for example, a CUseeMe Web invitation to the remote chatter. 
Windows hooks are installed in the IM applications in order to receive video buttons events as 
well as hooks on windows creations to check which IM or chat window is visible. 

The conference room in which the video conference takes place may or may not be an 
independent window, and by default it does not stick to the IM windows. The video is an 
autonomous communication tool, like chat or IM. Shortcuts are provided from within the IM 
window or the IM plugin UI, but if the IM window is closed, the video conference may persist. 

Two different layouts are provided, and a setting allows the user to switch between the 
layouts. The ConfRoom is a singleton resource, as long as a video conference is in progress, all 
the video shortcuts in all the IM windows are grayed. An option allows the user to stick its video 
if that mode is preferred. CUseeMe for IPulse is an example of this kind of integration and an 
example of configurable layout. 

FIG. 8 illustrates the event sequence between the IM plugin in accordance with the 
invention and a remote buddy who does not have the plugin. Back-end Local represents the 
interception of the IM protocol. The user interface on the local side contains the plugin buttons, 
IM client is the remote IM client and remote IM UI is the IM application's IM window. If the 
plugin download and installation are fast enough, the callee might call back the caller before the 
timeout. 

FIG. 9 illustrates the event sequence between two plugins. Back-end Local and Back-end 
remote represent the interception of the IM protocol. Im plugin UI on both sides contains the 
plugin buttons. 

2. Application APIs 

This approach operates in an analogous fashion to the proxy and plugin described above. 
But, instead of intercepting messages with a proxy, the vendor has supplied APIs for a clean 



CSM-0002 



28 



integration into its EVt product. The plugin of the invention can register for events and query 
presence information of buddies in the list. The plugin can also create new message channels for 
communicating conference rendezvous information between endpoints, as well as intercept 
messages upon receipt. In addition, the application may also provide APIs such that the plugin of 
the invention can add menu choices and/or buttons to the IM application's user interface. 
Otherwise, the actual message content, sequence and parameters are the same as those used in the 
proxy approach. FIG. 10 is a schematic block diagram illustrating the API approach. 

While this invention has been particularly shown and described with references to 
preferred embodiments thereof, it will be understood by those skilled in the art that various 
changes in form and details may be made herein without departing from the spirit and scope of 
the invention as defined by the following claims. 
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